Image processing apparatus and method thereof

ABSTRACT

An image processing apparatus having an image sensing unit and a subject detection unit for performing a process of detecting a subject in an image inputted from said image sensing unit, has a detection size operation unit that sets a resolution of a target object to be detected by the subject detection unit, and an image conversion unit that converts a resolution of the input image, based on the resolution of the target object set by the detection size operation unit. The subject detection unit performs the process of detecting the subject in the image whose resolution has been converted by the image conversion unit.

CROSS REFERENCE TO RELATED APPLICATIONS

This is a continuation of co-pending U.S. application Ser. No.11/469,115 filed Aug. 31, 2006, which claims the benefit of JapanesePatent Application No. 2005-256859 filed on Sep. 5, 2005, both of whichare hereby incorporated by reference herein in their entirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing apparatus and amethod thereof, and more particularly to an image processing apparatusand a method thereof for detecting a predetermined target object(subject) from an image inputted from an image input device such as adigital camera or the like.

2. Description of the Related Art

Conventionally, a digital camera or a digital video for detecting aspecific subject such as an individual or a face in an input image andperforming a process which is suitable for the detected subject, hasbeen proposed.

In Japanese Patent No. 3164692, a camera has been proposed whichincludes an individual recognition unit for recognizing that a subjectis an individual, and a distance detection unit for detecting a distanceto the subject. This camera includes a unit for adjusting a focallength, a focal position and an aperture based on the detected distanceto the subject, and controlling an entire face of the individual to beapproximately within a depth of field.

In Japanese Patent Application Laid-Open No. 2001-309225, a camera hasbeen proposed which detects more than one face of persons included in animage in order to improve the quality of the image.

In Japanese Patent Application Laid-Open No. 2003-107555, an imagesensing apparatus has been proposed which has a face detection unit fordetecting a face of an individual from image data, and controls anexposure based on a result of the detection. This image sensingapparatus includes a photometry unit for performing photometry withrespect to a photometry area which is set to the individual's facedetected by the face detection unit, and an exposure control unit forcalculating an exposure amount based on a result of the photometry ofthe individual's face and performing an exposure control based on thecalculated exposure amount.

As a face detection processing method for detecting the face in theimage, various methods have been proposed.

For example, in “Rapid object Detection using a Boosted Cascade ofSimple Features”, P. Viola, M. Jones, Proc. of IEEE Conf. CVPR, 1, pp.511-518, 2001, a high speed face detection method has been proposed.Moreover, in “Convolutional Spiking Neural Network Model for Robust FaceDetection”, M. Matsugu, K. Mori, et. al, 2002, International ConferenceOn Neural Information Processing (ICONIP02) and “Neural Network-BasedFace Detection”, H. A. Rowley, S. Baluja, T. Kanade, 1996, ComputerVision and Pattern Recognition (CVPR '96), neural networks forperforming a face detection have been proposed.

In the case of sensing the image with a digital camera, where, how largeand how many faces exist in the image vary substantially depending on asensing condition. Thus, it is required that the face detectionprocessing method mounted on the digital camera does not depend on aposition, the size or the number of faces in the image.

A basic concept of detecting a specific pattern such as the face or thelike (hereinafter referred to as “detection pattern”) from the image isas follows. First, an area of a specific size is clipped from the image,a feature of the area and a feature of the detection pattern arecompared and investigated. If those features are similar, it isdetermined that the clipped area is the detection pattern and thespecific pattern such as the face or the like exists in the area.

Thus, as shown in FIG. 14, by clipping the area of a specific sizesequentially from an image 1401 and investigating those clipped areasrespectively, it is possible to perform a detection independent of theposition and the number of faces in the input image.

Moreover, in order to perform a detection independent of the size of aface, as shown in FIG. 15, a plurality of images obtained by convertinga resolution of the input image discretely, called pyramid images, areprepared, and the areas are clipped and investigated from the images ofrespective resolutions.

Moreover, it is also possible to prepare a plurality of types ofdetection patterns of specific different sizes and perform thedetection, without performing the resolution conversion with respect tothe image.

A clipping position, each displacement, that is to say, a positionalinterval due to the resolution conversion, and an interval of theresolutions depend on positional robustness and size robustness of eachdetection method. For example, in the case where the pattern detectionis performed with respect to one clipped area and the detection patternis not detected unless the detection pattern exists in the center of thearea, it is necessary to move the clipping position pixel by pixel.Here, if the method has the positional robustness of ±2 pixels, it ispossible to move the clipping position by 5 pixels. In other words, itis possible to reduce the amount of calculation. Also, similarly, interms of the size, in a detection method having double size robustnesswith respect to one area, if images of two resolutions of 1/1 times and1/2 times are prepared, it is possible to accept a quadruple sizevariation. On the other hand, in a detection method having the sizerobustness of about 1.4 times, it is necessary to prepare the images offour resolutions of 1/1 times, 1/√2 times, 1/2 times and 1/(2√2) timesand the calculation amount increases.

Generally, however, if the robustness is raised, detection precisiontends to be reduced. In other words, even when the detection patternsuch as the face or the like exists in the image, the detection patternmay not be detected, or a pattern such as a background or the like whichis completely different from the detection pattern may be detectedincorrectly as the detection pattern. If such a detection error occurs,when the exposure is controlled in accordance with the face in the imageas proposed in Japanese Patent Application Laid-Open No. 2003-107555,the exposure may be controlled in accordance with the area other thanthe face which has been incorrectly detected, for example, thebackground or the like.

Conversely, if the robustness is reduced, it is necessary to thicken aclipping positional interval and the interval of the resolutions, andthe calculation amount increases in this case. When the calculationamount increases, it may take time to perform the detection process withrespect to the image.

SUMMARY OF THE INVENTION

The present invention has been made in consideration of the abovesituation, and has as its object to facilitate a setting for performinga detection with a detection performance which is necessary and enoughfor the user, in detecting a specific pattern in a image.

According to the present invention, the foregoing object is attained byproviding an image processing apparatus having an image input unit and adetection unit for performing a process of detecting a predeterminedtarget object in an image inputted from the image input unit,comprising: a size changing unit that sets a resolution of the targetobject to be detected by the detection unit; and a resolution conversionunit that converts a resolution of the input image based on theresolution of the target object set by the size changing unit, whereinthe detection unit performs the process of detecting the target objectin the image whose resolution has been converted by the resolutionconversion unit.

According to the present invention, the foregoing object is alsoattained by providing an image processing method comprising: an imageinput step; a detection step of performing a process of detecting apredetermined target object in an image inputted at the image inputstep; a size changing step of setting a resolution of the target objectto be detected at the detection step; and a resolution conversion stepof converting a resolution of the input image, based on the resolutionof the target object set at the size changing step, wherein thedetection step performs the process of detecting the target object inthe image whose resolution has been converted at the resolutionconversion step.

According to another aspect of the present invention, the foregoingobject is also attained by providing an image processing apparatushaving an image input unit and a detection unit for performing a processof detecting a predetermined target object in an image inputted from theimage input unit, comprising: a size changing unit that sets a size ofthe target object to be detected by the detection unit; a processingtime acquisition unit that acquires a predicted processing time which isrequired for detecting the target object by the detection unit, based onthe size of the target object set by the size changing unit; and anotification unit that notifies the predicted processing time acquiredby the processing time acquisition unit.

Furthermore, the foregoing object is also attained by providing An imageprocessing method comprising: an image input step; a detection step ofperforming a process of detecting a predetermined target object in animage inputted at the image input step; a size changing step of settinga size of the target object to be detected at the detection step; aprocessing time acquisition step of acquiring a predicted processingtime which is required for detecting the target object at the detectionstep, based on the size of the target object set at the size changingstep; and a notification step of notifying the predicted processing timeacquired at the processing time acquisition step.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate embodiments of the invention and,together with the description, serve to explain the principles of theinvention.

FIG. 1 is a block diagram showing a schematic functional configurationof an image sensing apparatus according to a first embodiment of thepresent invention;

FIG. 2 is a flowchart showing a subject detection operation according tothe first embodiment of the present invention;

FIG. 3 is a rear view of the image sensing apparatus according to thefirst embodiment of the present invention;

FIGS. 4A and 4B are diagrams for illustrating a resolution conversion;

FIG. 5 is a diagram showing a display example of a result of subjectdetection according to the first embodiment of the present invention;

FIG. 6 is a diagram showing a display for confirming a setting accordingto the first embodiment of the present invention;

FIGS. 7A and 7B are diagrams showing transition of the display forconfirming the setting associated with an operation by a detection sizeoperation unit;

FIGS. 8A and 8B are diagrams showing the displays at the time of asubject detection non-execution mode;

FIG. 9 is a flowchart for illustrating a resolution conversion processaccording to a second embodiment of the present invention;

FIG. 10 is a rear view of an image sensing apparatus according to thesecond embodiment of the present invention;

FIG. 11 is a block diagram showing a schematic functional configurationof an image sensing apparatus according to a third embodiment of thepresent invention;

FIG. 12 is a flowchart showing the subject detection operation accordingto the third embodiment of the present invention;

FIG. 13 is a flowchart for illustrating a coupled zoom control processaccording to the third embodiment of the present invention;

FIG. 14 is a diagram for illustrating conventional clipped areas; and

FIG. 15 is a diagram for illustrating conventional pyramid images.

DESCRIPTION OF THE EMBODIMENTS

Preferred embodiments of the present invention will be described indetail in accordance with the accompanying drawings. However, thedimensions, shapes and relative positions of the constituent parts shownin the embodiments should be changed as convenient depending on variousconditions and on the structure of the apparatus adapted to theinvention, and the invention is not limited to the embodiments describedherein.

First Embodiment

FIG. 1 is a diagram showing a configuration of an image sensingapparatus according to the first embodiment. In FIG. 1, referencenumeral 101 denotes a control unit; 102, an image sensing unit; 103, asubject detection unit; 104, a subject detection memory; 105, an imageconversion unit; 106, a detection size operation unit; 107, a detectionsize storage unit; 108, a processing time calculation unit; 109, asetting confirmation switch; 110, a shutter switch; 111, a memory; 112,a display circuit; and 113, a display unit.

The control unit 101 is a unit for controlling the entire image sensingapparatus. To the control unit 101, the image sensing unit 102, thesubject detection unit 103, the subject detection memory 104, the imageconversion unit 105, the detection size operation unit 106, thedetection size storage unit 107, the processing time calculation unit108, the setting confirmation switch 109, the shutter switch 110, thememory 111, and the display circuit 112 are connected. The control unit101 controls so that each unit operates at an appropriate timing.

The image sensing unit 102 comprises an image sensing lens 120, an imagesensor 121 for converting a subject optical image which has passedthrough the image sensing lens 120 into an analog electrical signal, anA/D converter 122 for converting the converted analog electrical signalinto a digital signal, and an image processing circuit 123. The imageprocessing circuit 123 creates an appropriate image signal by performinga predetermined processes, such as a white balance correction, a gammaconversion or the like, on the digital signal obtained by the A/Dconversion performed by the A/D converter 122. It should be noted thatthe image sensing unit 102 performs an image sensing process based on acontrol signal from the control unit 101.

The memory 111 stores programs for the control unit 101 and data usedfor control. Moreover, the memory 111 also stores an image sensed by theimage sensing unit 102.

The detection size operation unit 106 sets a resolution (hereinafterreferred to as “detection size”) of a specific pattern (subject) that isa detection object, with respect to a resolution of an input image, in adetection process by the subject detection unit 103. The detection sizestorage unit 107 stores this detection size.

The image conversion unit 105 converts the resolution of an image sensedby the image sensing unit 102 so that it may become a resolution atwhich the subject of the detection size stored in the detection sizestorage unit 107 can be detected, as will be described below. The imagewhose resolution has been converted by the image conversion unit 105 isstored in the subject detection memory 104. The subject detection unit103 performs the detection process of detecting the subject on theresolution-converted image stored in the subject detection memory 104.

The processing time calculation unit 108 calculates an approximateprocessing time related to the detection process performed by thesubject detection unit 103, based on the detection size set by thedetection size operation unit 106.

The setting confirmation switch 109 is a switch for switching to displayintelligibly the detection size stored in the detection size storageunit 107. If the setting confirmation switch 109 is On, a frame of thesize and the subject of CG (Computer Graphics) as will be describedbelow are displayed in a superimposed manner. If the settingconfirmation switch 109 is Off, these are not displayed.

The display circuit 112 controls the display unit 113 to perform adesired screen display. Displayed on the display unit 113 is informationuseful for the image sensing, such as the image sensed by the imagesensing unit 102, the image stored in the memory 111, the subjectdetected by the subject detection unit 103, the detection size stored inthe detection size storage unit 107, and the like.

FIG. 3 shows an example of a rear side of the image sensing apparatusaccording to the first embodiment. In FIG. 3, reference numeral 131denotes a detection size display bar, and reference numeral 132 denotesa processing time display bar. Those bars are displayed within thedisplay unit 113.

The detection size display bar 131 displays the detection size stored inthe detection size storage unit 107. Moreover, the processing timedisplay bar 132 displays the approximate processing time obtained by theprocessing time calculation unit 108 based on the detection size storedin the detection size storage unit 107. This calculation method will bedescribed below.

Next, with reference to a flowchart of FIG. 2, a subject detectionoperation in the image sensing apparatus according to the firstembodiment will be described. The process shown in FIG. 2 is basicallycontrolled by the control unit 101.

First, when an image sensing mode is set by a user at step S11, afunction of an Electrical View Finder (EVF) is started at step S12. TheEVF function is realized by sensing images periodically at intervals ofpredetermined time by the image sensing unit 102 to generate a movingimage, and displaying the images on the display unit 113 via the displaycircuit 112.

At step S13, it is determined whether or not where SW1 is On, which is acondition where an image sensing preparation has been instructed inresponse to, for example, a half stroke of the shutter switch 110. IfSW1 is On, the process proceeds to step S14. If SW1 is Off, the processproceeds to step S20.

At step S14, the image which has been sensed by the image sensing unit102 when SW1 has become On or immediately after then, is stored in thememory 111, and concurrently in the image conversion unit 105, theresolution of the input image is converted based on the detection sizestored in the detection size storage unit 107. The resolution-convertedimage is stored in the subject detection memory 104.

Here, the resolution conversion process performed in the imageconversion unit 105 at step S14 will be described.

It should be noted that, in this resolution conversion process, unlessotherwise noted, “the resolution is set to 1/2” means that theresolution is set to 1/2 both in the horizontal direction and thevertical direction. For example, if the resolution of an image of aresolution of 640×480 pixels as shown in FIG. 4A is changed to 1/2, thisimage becomes an image of a resolution of 320×240 pixels as shown inFIG. 4B.

As described in the background of the invention, there are manydetection processing methods of detecting the subject in the image.However, in every method, there is a minimum limit in the resolution fordetecting an object. This minimum resolution depends on each processingmethod, and in a method described in “Neural Network-Based FaceDetection”, H. A. Rowley, S. Baluja, T. Kanade, 1996, Computer Visionand Pattern Recognition (CVPR '96), for example, a minimum size(resolution) of a detectable face is 20×20 pixels.

Moreover, a detection processing time depends on the resolution of theimage subjected to the detection. If the resolution of the imagesubjected to the detection is high, the processing time increasesbecause the number of clipped areas increases.

Therefore, at step S14, the resolution of the input image is convertedso that the detection size stored in the detection size storage unit 107may become the minimum resolution of the detection method to be used.

Here, let the minimum resolution of the detection method be Q, aresolution conversion ratio be R, and the detection size stored in thedetection size storage unit 107 be S. Then, the resolution conversionratio R is given by:

R=Q/S  (1)

Next, let the converted resolution be A and the resolution before theconversion be B, then the resolution A of the image after resolutionconversion is given by:

A=B×R  (2)

In other words, if the resolution of the input image is 640×480 pixels,the detection size is 60×60 pixels, and the minimum resolution is 20×20pixels,

R=20/60=1/3  (3)

A=640×(1/3)≈214 (pixels) (horizontal direction)  (4)

A=480×(1/3)=160 (pixels) (vertical direction)  (5)

In other words, in this case, the resolution of the input image of theresolution of 640×480 pixels is converted into 214×160 pixels. As aresolution conversion method, a general method using a linearinterpolation or Bicubic can be used. In consideration of an imagequality and the processing time, typically the linear interpolation isused.

Further continuously, by using this converted image, pyramid images(images of a plurality of different resolutions) are created asdescribed in the background of the invention with reference to FIG. 15.For example, if a detection method having double size robustness isused, the resolution of the image converted into the 214×160 pixels asshown above is converted into 107×80 pixels. Furthermore, the resolutionof the image is converted into 54×40 pixels, and into 27×20 pixels. Byperforming the detection process at step S16 for all of theseresolution-converted images, the size of the detected subject maycorrespond to 60×60 pixels to 480×480 pixels in the input image of theresolution of 640×480 pixels.

It should be noted that in the resolution conversion, when theresolution is converted into the minimum resolution at which the subjectcan be detected, there may be the case where a detection rate decreases.Therefore, the resolution conversion rate R of the resolution conversionto be performed first may be set as follows:

R=Q/S+α  (6)

α may be an appropriate positive number. For example, in the abovedescribed example, when α is set to 1/15 and R=1/2.5, instead of R=1/3,the resolution of the input image of 640×480 pixels becomes 256×192pixels after the resolution conversion.

When the above described process at step S14 is completed, the processproceeds to step S15. At step S15, one of the pyramid images stored inthe subject detection memory 104 is selected. Among the images to whichthe detection process has not been performed, the image of the highestresolution or the image of the lowest resolution is selected.

At step S16, the detection process is performed with respect to theimage selected at step S15. As introduced in the background of theinvention, many detection processing methods have been proposed, and anymethod may be applied. When the detection process is completed, thenumber of the subjects in the image, and a position and a size of eachsubject in the image are obtained as a detection result. It should benoted that the detected position is obtained as a subject position inthe input image (in the above described example, the image of 640×480pixels), by performing a correction associated with the resolutionconversion process, with respect to detected coordinates values whichhave been obtained in the image of that resolution. Also, the size ofthe subject is similarly applied with the correction associated with theresolution conversion and obtained as the size of the subject occupyingin the input image.

At step S17, it is determined whether the detection process has beenperformed for the images of all of the resolutions stored in the subjectdetection memory 104. If the detection process has not been completed,the process returns to step S15 to repeat the above described processes,the image of the next resolution is selected and the detection processis performed. If the detection process has been performed for the imagesof all of the resolutions, the process proceeds to step S18.

At step S18, the detection results obtained with respect to the imagesof all of the resolutions stored in the subject detection memory 104 areintegrated. The integration is performed based on the positions and thesizes in the input image. In other words, when there are a plurality ofdetection results at near positions and the sizes of these detectionresults are nearly equal, these detection results are integrated as anidentical subject. An average position and an average size of theplurality of detection results are taken as the position and the size ofthe integrated subject. However, if a plurality of detection resultshave been obtained in which the positions are near but the sizes aredifferent, the detection size stored in the detection size storage unit107 is prioritized, and the detection result which is nearest to thedetection size is selected.

At step S19, the image stored in the memory 111 is displayed on thedisplay unit 113 via the display circuit 112. Also concurrently, basedon the subject detection result integrated at step S18, a detectionframe of the size of each subject is displayed at the position of eachsubject in the image displayed on the display unit 113.

As long as the condition where SW1 is On continues, the process of theabove described steps S14-S19 is repeated. In other words, while in thecondition where SW1 is On, the processing time of a subject detectionprocess decides a display rate of the EVF. In this way, by reducing theprocessing time as much as possible in the subject detection, usabilityfor the user improves.

FIG. 5 shows a schematic diagram of the image displayed on the displayunit 113 when the subject is detected in the condition where SW1 is On.In FIG. 5, reference numerals 141 and 142 denote subjects, and referencenumerals 143 and 144 denote the detection frames for the detectedsubjects.

On the other hand, at step S13, if SW1 is not On, the process proceedsto step S20.

At step S20, it is determined whether the setting confirmation switch109 is On or Off. If the setting confirmation switch 109 is On, theprocess proceeds to step S21. At step S21, the detection size stored inthe detection size storage unit 107 is obtained. As shown in FIG. 6,together with the detection size display bar 131 and the processing timedisplay bar 132, a detection size confirmation frame 161 correspondingto the obtained detection size is displayed on the display unit 113. Itshould be noted that, since this confirmation frame 161 is displayed sothat the user can easily understand the size of the subject to bedetected, if the subject is a face for example, the CG of the face andthe like may be displayed instead of the confirmation frame 161.

At step S22, with an operation on the detection size operation unit 106by the user, it is determined whether a change in the detection size hasbeen instructed. If the change has not been instructed, the processreturns to step S13 and the above described processes are repeated. Ifthe change has been instructed, the process proceeds to step S23.

At step S23, the detection size stored in the detection size storageunit 107 is obtained. If the detection size operation unit 106 has beenoperated to enlarge the size, a predetermined value is added to thedetection size. If the detection size operation unit 106 has beenoperated to reduce the size, the predetermined value is subtracted fromthe detection size. At this time, the addition or the subtraction isperformed such that the detection size falls within a predeterminedrange prescribed by a detection performance of the adopted detectionprocessing method. In other words, as described above, the detectionwith the detection size of the resolution which is less than or equal tothe minimum limit of the adopted detection processing method (in theabove described example, less than 20×20 pixels), or of the resolutionwhich is more than the resolution of the input image (in the abovedescribed example, more than 480×480 pixels) is impossible. Thus, thedetection size is limited to fall within the predetermined range of adetectable size. Specifically, if it is instructed to reduce the sizewhen the set detection size is 20×20 pixels, and if it is instructed toenlarge the size when the set detection size is 480×480 pixels, theinstruction is not executed.

At step S24, the changed detection size is stored in the detection sizestorage unit 107.

Subsequently at step S25, based on the detection size which has beennewly stored in the detection size storage unit 107, the display of thedetection size display bar 131 is updated. Furthermore, the approximateprocessing time at the detection size is calculated in the processingtime calculation unit 108, and the display of the processing timedisplay bar 132 is updated. In other words, when the detection size ischanged, the detection size display bar 131 is changed, and inconjunction with it, the display of the processing time display bar 132is changed. Thus, it is possible for the user to set a desired detectionsize and understand a rough indication of the processing time with thatdetection size. Therefore, it is also possible to change the detectionsize in order to set the desired processing time. It should be notedthat if the setting confirmation switch is On, the size of the detectionsize confirmation frame 161 is updated and displayed depending on thechanged detection size at step S25.

Next, the calculation method for the approximate processing time will bedescribed below.

As described in the background of the invention, generally the detectionprocess is performed by clipping an area of a specific size from theimage, comparing and investigating a feature of the pattern of the areaand a feature of an object pattern. If the detection processing time perone clipped area is constant, the total detection processing time is inproportion to the number of clipped areas. In other words, the totaldetection processing time is in proportion to the resolution of theimage converted at step S14. However, among the detection processingmethods, there are a method of first checking a frequency of eachclipped area, and if the frequency of the clipped area is low,determining that the area does not include a desired subject. Further,as a similar method, there is a method of cascade-connecting weakdiscriminators, in which it takes more processing time if the area issimilar to the desired subject, and the like. In these methods, itcannot be said that the detection processing time per one area isconstant for all of the clipped areas. However, when an average value ofthe processing time per one area is obtained by using a large amount of,that is, tens of thousands of images, the average value becomes a valuein which even a distribution of existences of low frequency areas or theareas similar to the subject in those images has been considered. Theprocessing time predicted from the value which has been obtained byusing a large amount of data in this way, shows a precision which isenough to be used as the rough indication of the processing time. Inthis way, it is possible to predict a rough processing time from theaverage value of the processing time per one area which has beenobtained from the large amount of data, and the number of the areaswhich can be clipped from the converted image.

Moreover, as shown in the background of the invention, depending onpositional robustness of the adopted detection method, an amount ofdisplacement of a clipping position varies.

From the above description, let the minimum resolution of the useddetection method be Q, the resolution of the image after the resolutionconversion be Bs, the average value of the processing time per one areabe C, the displacement amount of the clipping position be D, and therough processing time per one image after the resolution conversion beTs, then the rough processing time Ts is given by:

Ts=C×(Bs/D−Q+1)  (7)

For example, when the minimum resolution Q=20×20 pixels, the resolutionof the converted image Bs=256×192 pixels, the average value of theprocessing time per one area C=1 μsec, and the displacement amount D is2, the rough processing time Ts is given as:

Ts=1 μsec×{(256/2−20+1)×(192/2−20+1)}≈84 msec  (8)

Since the rough processing time Ts as shown above is the processing timewith respect to the image of one resolution among the images of theplurality of different resolutions, it is possible to estimate the roughprocessing time with respect to the images of all of the resolutions byadding the processing time for the image of each resolution. In otherwords, a rough processing time T taken for the detection process withrespect to all of the images of the plurality of resolutions is given bythe following expression (9):

T=ΣTs  (9)

It should be noted that the minimum resolution of the detection methodQ, the average value C of the processing time per one area, and thedisplacement amount D of the clipping position depend on the detectionmethod and are known. Therefore, by providing a table for obtaining theresolution Bs of the image after the resolution conversion with thedetection size, the input image, and the resolution conversion ratio forcreating the pyramid images being input values and the detectionprocessing time being an output value, it is possible to obtain therough processing T can be searched from this table. This makes itunnecessary for the processing time calculation unit 108 to calculatethe above calculation each time.

FIGS. 7A and 7B show changes in the display on the display unit 113 whenthe operation by the detection size operation unit 106 is performed. Itshould be noted that the setting confirmation switch 109 is set to On,and the detection size which is currently set is displayed as theconfirmation frame 161 in FIG. 7A. In contrast to FIG. 7A, FIG. 7B showsa result of operating the detection size operation unit 106 to reducethe detection size. In FIG. 7B, a tab mark indicating the currentdetection size on the detection size display bar 131 has moved toward asmall value (Small), and a tab mark indicating the current processingtime on the processing time display bar 132 has moved to increase theprocessing time (Slow). FIG. 7B further shows that the size of theconfirmation frame 161 has become small. In this way, by having thedetection size display bar 131 and the processing time display bar 132operate in conjunction with each other, it is possible to easilyunderstand that when the detection size is changed, also the processingtime is changed.

According to the first embodiment as described above, by operating thedetection size operation unit 106 to specify the detection size by theuser, it is possible to realize reducing the processing time whiledetecting the subject of the size to be detected. Moreover, bydisplaying the detection size in a two-dimensional way, a user caneasily understand the detection size, and thereby to set the detectionsize to a desired size. Furthermore, it is possible to set the detectionsize within a range in which the subject can be detected. Moreover,since the detection size and the processing time are changed inconjunction with each other, a user can easily understand a relationbetween the detection size and the processing time.

Although it has been described that the pyramid images are created inthe resolution conversion process at step S14 in FIG. 2, it should benoted that the image may be converted into only one resolution, insteadof the images of the plurality of different resolutions. In other words,the image may be converted based on the detection size stored in thedetection size storage unit 107, and after that, the image of adifferent resolution may not be created.

As described above, when the detection process is performed by using theimage of only one resolution, it is not possible to detect the subjectof a size different from the detection size stored in the detection sizestorage unit 107, beyond the size robustness of the adopted detectionprocessing method. However, as the detection processing time is reduced,it is also possible to reduce erroneous detections.

Moreover, although the detection size is limited to fall within thedetectable predetermined range at step S23 in FIG. 2, this limitationmay not be performed. In that case, the subject detection process maynot be performed when the changed detection size becomes out of thepredetermined range, and that effect may be displayed. FIGS. 8A and 8Bshow display examples at this time. FIG. 8A shows the time when the tabmark indicating the detection size displayed on the detection sizedisplay bar 131 is at the smallest position. If the detection sizeoperation unit 106 is operated for more than or equal to a predeterminedtime to further reduce the detection size, the process shifts to asubject detection non-execution mode in which the subject detectionprocess is not performed. When it is in the subject detectionnon-execution mode, as shown in FIG. 8B, the detection size display bar131 and the processing time display bar 132 are not displayed on thedisplay unit 113, and a display 181 for indicating the subject detectionnon-execution mode is performed.

Second Embodiment

Next, the second embodiment of the present invention will be described.

In this second embodiment, the resolution conversion, which is performedby the image conversion unit 105 in the first embodiment, is performedwith a fixed resolution. In other words, mainly, in this secondembodiment, the process at step S14 of the flowchart shown in FIG. 2 isdifferent from that of the first embodiment. Since other processes andthe configuration of the image sensing apparatus are similar to thosedescribed in the first embodiment, the description thereof is omittedhere.

Hereinafter, the resolution conversion process performed at step S14 inthe second embodiment will be described with reference to a flowchartshown in FIG. 9.

In the second embodiment, at step S14, three processes of creating thepyramid images, calculating a necessary detection resolution, andstoring an image selection are performed.

First, at step S31, the resolution conversion ratio is set based on thesize robustness of the adopted detection method. The resolution of theinput image is converted sequentially with that conversion ratio, andthe pyramid images are created.

Here, let the resolution conversion ratio be Rf, the resolution afterthe conversion be A, and the resolution before the conversion be B, thenthe resolution A is given by:

A=B×Rf  (10)

This resolution conversion is repeated until the minimum resolution ofthe detection method to create the pyramid images.

For example, if the resolution of the input image is 640×480 pixels, thesize robustness of the detection method is double, and the minimumresolution is 20×20 pixels, the resolution A becomes as follows:

Rf=1/2  (11)

A=640×(1/2)=320 (pixels) (horizontal direction)  (12)

A=480×(1/2)=240 (pixels) (vertical direction)  (13)

Then the images of 160×120 pixels, 80×60 pixels and 40×30 pixels arecreated in turn.

Next, at step S32, the resolution required for detecting the subject ofthe detection size stored in the detection size storage unit 107, thatis, the necessary detection resolution is obtained. To obtain thisresolution, the same method as the method shown in the first embodimentis used.

For example, let the resolution conversion ratio be R, the necessarydetection resolution be An, and the resolution of the input image is640×480 pixels, the detection size is 60×60 pixels, and the minimumresolution is 20×20 pixels, the necessary detection resolution Anbecomes:

R=20/60=1/3  (14)

An=640×(1/3)≈214 (pixels)(horizontal direction)  (15)

An=480×(1/3)=160 (pixels)(vertical direction)  (16)

In other words, in this case, 214×160 pixels is the necessary detectionresolution.

Next, at step S33, the image to be used for the detection is selected,and stored in the subject detection memory 104. Here, among the pyramidimages created at step S31, the image of the lowest resolution which ishigher than the necessary detection resolution obtained at step S32. Forexample, in the above described example, 320×240 pixels are selected.

All of the images of the resolutions which are lower than the selectedimage are stored in the subject detection memory 104. For example, inthe above described example, images of 320×240 pixels, 160×120 pixels,80×60 pixels and 40×30 pixels are stored.

FIG. 10 shows the rear side of the image sensing apparatus according tothe second embodiment. In the second embodiment, when the detection sizeis changed, the resolution of the image to be used for the detectiondiscretely changes. Thus, the processing time also discretely changes.Therefore, as shown in FIG. 10, a processing time display bar 232displayed on the display unit 113 is different from that of the firstembodiment, and performs a discrete display.

As described above, according to the second embodiment, by performingthe resolution conversion process with a fixed conversion ratio, thereis an advantage that in the case where the converted image is createdusing a circuit, it becomes easy to design the circuit.

Third Embodiment

Next, the third embodiment of the present invention will be described.

In the third embodiment, if a zoom control is performed, the detectionsize is automatically changed in proportion to its zoom amount.

FIG. 11 is a block diagram showing the configuration of an image sensingapparatus according to the third embodiment. In FIG. 11, theconfiguration similar to FIG. 1 is referred to by the same referencenumeral, and the description thereof is omitted. In comparison to theconfiguration shown in FIG. 1, a zoom operation unit 314 has been added.By operating this zoom operation unit 314, the image sensing lens 120 inthe image sensing unit 102 is controlled to change a zoom magnification.

Next, with reference to a flowchart of FIG. 12, the subject detectionoperation including the operation on the zoom operation unit 314 will bedescribed. The process shown in FIG. 12 is basically controlled by thecontrol unit 301. It should be noted that in the processes shown in FIG.12, the similar process as the process described with respect to FIG. 2is referred to by the same reference numeral, and the descriptionthereof is omitted.

At step S41, it is determined whether the user has operated the zoomoperation unit 314. If it is determined that the user has not operatedit, the process proceeds to step S22, and then the process as describedin FIG. 2 is performed. On the other hand, if it is determined that theuser has operated the zoom operation unit 314, the process proceeds tostep S42, and a coupled zoom control is performed.

FIG. 13 is a flowchart showing the coupled zoom control processperformed at step S42.

At step S51, the zoom amount is set according to the operation on thezoom operation unit 314 by the user, and a zoom lens within the imagesensing lens 120 of the image sensing unit 102 is controlled.

Next, at step S52, it is determined whether or not a coupled zoom modehas been set. Here the coupled zoom mode refers to a mode of couplingthe zoom control and the detection size. Here the coupled zoom mode isset with a set button not shown, or is set when the subject detectionhas been performed immediately before. A condition of “the subjectdetection has been performed immediately before” refers to a conditionwhere SW1 has been set to On and the subject has been detected with theprocess at steps S14 to S19, within a predetermined time when the zoomoperation unit 314 is operated. When the user performs the zoom controlimmediately after the subject has been detected, it is considered thatthe user intends to change the size of the subject in the image. Thus,in the third embodiment, if “the subject detection has been performedimmediately before”, the coupled zoom mode is set.

If the coupled zoom mode is not set, since it is unnecessary to performthe process of more than controlling the zoom lens, the process returnsto the flowchart of FIG. 12.

On the other hand, if the coupled zoom mode has been set, the processproceeds to step S53. At step S53, a new detection size is obtained fromthe detection size stored in the detection size storage unit 107 and azoom variation. Let the detection size stored in the detection sizestorage unit 107 before the zoom control be So, the zoom variation beZv, and the new detection size be Sn, then the new detection size Sn isgiven by the following equation (17):

Sn=So×Zv  (17)

For example, when the detection size which has been stored in thedetection size storage unit 107 is 48×48 pixels, and a zoom variationratio is 2.0 times, the new detection size becomes 96×96 pixels. Next,at step S54, the changed detection size is stored in the detection sizestorage unit 107 again.

At step S55, similarly to the first embodiment, the display on thedetection size display bar 131 is changed according to the detectionsize stored in the detection size storage unit 107. Moreover, theprocessing time at that detection size is predicted by calculation orusing the table, and the display on the processing time display bar 132is changed.

In this way, in the third embodiment, if the coupled zoom mode has beenset, such as if the user has specified or if the subject detection hasbeen performed immediately before the zoom control, the detection sizeis automatically changed in proportion to a zoom control amount.According to this control, there is an advantage that it becomesunnecessary to change the detection size after the zoom control iscompleted.

Although the case where the zoom is an optical zoom has been describedin the above described third embodiment, it should be noted that it isapparent that the present invention is applicable even to the case of anelectronic zoom and the case where both the optical zoom and theelectronic zoom are used.

Moreover, although the case where the present invention is applied tothe image sensing apparatus has been described in the above describedfirst to third embodiments, it is also possible to apply the presentinvention to an apparatus for inputting the image externally anddetecting the subject in the input image.

Other Embodiments

The invention can be implemented by supplying a software program, whichimplements the functions of the foregoing embodiments, directly orindirectly to a system or apparatus, reading the supplied program codewith a computer of the system or apparatus, and then executing theprogram code. In this case, so long as the system or apparatus has thefunctions of the program, the mode of implementation need not rely upona program.

Accordingly, since the functions of the present invention areimplemented by computer, the program code installed in the computer alsoimplements the present invention. In other words, the claims of thepresent invention also cover a computer program for the purpose ofimplementing the functions of the present invention.

In this case, so long as the system or apparatus has the functions ofthe program, the program may be executed in any form, such as an objectcode, a program executed by an interpreter, or scrip data supplied to anoperating system.

Example of storage media that can be used for supplying the program area floppy disk, a hard disk, an optical disk, a magneto-optical disk, aCD-ROM, a CD-R, a CD-RW, a magnetic tape, a non-volatile type memorycard, a ROM, and a DVD (DVD-ROM and a DVD-R).

As for the method of supplying the program, a client computer can beconnected to a website on the Internet using a browser of the clientcomputer, and the computer program of the present invention or anautomatically-installable compressed file of the program can bedownloaded to a recording medium such as a hard disk. Further, theprogram of the present invention can be supplied by dividing the programcode constituting the program into a plurality of files and downloadingthe files from different websites. In other words, a WWW (World WideWeb) server that downloads, to multiple users, the program files thatimplement the functions of the present invention by computer is alsocovered by the claims of the present invention.

It is also possible to encrypt and store the program of the presentinvention on a storage medium such as a CD-ROM, distribute the storagemedium to users, allow users who meet certain requirements to downloaddecryption key information from a website via the Internet, and allowthese users to decrypt the encrypted program by using the keyinformation, whereby the program is installed in the user computer.

Besides the cases where the aforementioned functions according to theembodiments are implemented by executing the read program by computer,an operating system or the like running on the computer may perform allor a part of the actual processing so that the functions of theforegoing embodiments can be implemented by this processing.

Furthermore, after the program read from the storage medium is writtento a function expansion board inserted into the computer or to a memoryprovided in a function expansion unit connected to the computer, a CPUor the like mounted on the function expansion board or functionexpansion unit performs all or a part of the actual processing so thatthe functions of the foregoing embodiments can be implemented by thisprocessing.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

1. An image processing apparatus, comprising: an image input unitconfigured to input an image; a detection unit configured to performdetection processing of detecting a target object from the input imageby detecting a detection pattern that shows the target object and has alimit size in a used detection method from a detection image generatedfrom the input image input by the image input unit; a size setting unitconfigured to set a size of the detection pattern; a display unitconfigured to display the input image and the size of the detectionpattern set by the size setting unit in the input image; a size changingunit configured to change a resolution of the detection pattern whosesize is set by the size setting unit so that the detection pattern hasthe limit size; a conversion unit configured to generate a plurality ofthe detection images by converting a resolution of the input image to afirst resolution with a conversion rate used in changing the resolutionof the detection pattern by the size changing unit, and to a pluralityof resolutions lower than the first resolution with sizes of theresolution-converted input images not becoming smaller than the limitsize; and a calculation unit configured to calculate a predictedprocessing time to be required for the detection processing by thedetection unit based on the size set by the size setting unit, whereinthe detection unit is configured to perform the detection processing oneach of the plurality of detection images generated by the conversionunit and is configured to combine a plurality of acquired detectionresults, and wherein the display unit further is configured to displaythe predicted processing time.
 2. The image processing apparatusaccording to claim 1, wherein the calculation unit recalculates thepredicted processing time each time the size of the detection pattern isset by the size setting unit, and the display unit updates the displayof the predicted processing time to the recalculated predictedprocessing time.
 3. The image processing apparatus according to claim 1,further comprising: a zoom operation unit configured to change a zoomamount, wherein, when the zoom amount is changed, the size setting unitchanges the size of the detection pattern in proportion to the zoomamount.
 4. An image processing method, comprising: inputting an image;performing detection processing of detecting a target object from theinput image by detecting a detection pattern that shows the targetobject and has a limit size in a used detection method from a detectionimage generated from the input image; setting a size of the detectionpattern; displaying the input image and the set size of the detectionpattern in the input image; changing a resolution of the detectionpattern having the set size so that the detection pattern has the limitsize; generating a plurality of the detection images by converting aresolution of the input image to a first resolution with a conversionrate used in changing the resolution of the detection pattern, and to aplurality of resolutions lower than the first resolution with sizes ofthe resolution-converted input images not becoming smaller than thelimit size; calculating a predicted processing time to be required forthe detection processing based on the size set by the size setting unit;performing the detection processing on each of the plurality of thegenerated detection images and combining a plurality of acquireddetection results; and displaying the predicted processing time.